A digital camera includes an optical system that forms an image of a scene onto an array of detectors (i.e., sensors), referred to as a focal-plane array (FPA). Modern digital cameras typically contain several million to tens of millions of sensors producing images having millions of image pixels.
As pixel counts continue to increase and begin to approach a billion, it becomes increasingly difficult to image a scene with a single camera and FPA. As a result, large-scale imaging systems have been developed that utilize multiple camera systems, each of which images a different portion of a scene onto its respective FPA, thereby creating a plurality of digital sub-images. These sub-images are then stitched together using image processing to form a single high-pixel-count image.
This presents challenges for the image processing, however, because the individual sub-images must be registered with each other and blended together to form a substantially seamless composite image. Merging the sub-images requires several operations, including identifying corresponding features between sub-images and transforming one or more of the sub-images to coordinate with the overall image.
Most prior-art image stitching methods assume that the points in different sub-images are related by straight-forward coordinate transformations (i.e., homographies) between the homogeneous coordinate systems of each camera. After corresponding features have been identified between two cameras, a homography is fit between these features. Once these homographies are fit, the images are transformed so as to appear from the desired viewpoint of the composite image.
A-priori knowledge of the positioning of each camera within the arrangement facilitates image processing; however, the arrangement of the cameras used to generate the sub-images is subject to perturbations due to environmental factors, such as temperature, shock, vibration, etc. As a result, the registration between cameras can change, leading to variations that can be significant relative to the desired image resolution.
In addition, the focus of each individual camera is typically controllable. This gives rise to a wide range of angular scale among the sub-images, which makes it more difficult to determine the registration between the cameras. Further, the typical model assumes a pinhole camera. As a result, the effective “pinhole” location, or principal point position, and image angular scale of each microcamera depends on the focus of each microcamera. The homographies are dependent on the microcamera foci, however, and must be readjusted as the foci change.
Parallax between the cameras can further complicate the registration between sub-images. As a result, extensive analysis is often required to determine the relative positions of the sub-images and/or to deghost double images that can result from parallax. Parallax between overlapping regions of adjacent sub-images makes it impossible, in general, to overlap all features in the overlapping regions since the features will have different range-dependent displacements in the different cameras.
Stitching of multiple sub-images into a composite image, therefore, normally takes several tens of minutes to hours because the steps of finding features, corresponding them between images, and transforming and blending tens or hundreds of sub-images is a time consuming process. While such long times for imaging stitching can be tolerated in some applications, it is unacceptable in many other applications. In high-pixel-count video imagery, for example, tens or hundreds of sensor arrays simultaneously acquire image data at video rates of one to sixty frames per second or more. As a result, conventional image-stitching approaches are unsuitable.
A method for rapidly combining a plurality of sub-images into a single, composite image remains unrealized in the prior art.